1. Field of the Invention
The present invention generally relates to refreshed image displays from digitally transmitted data and, more particularly, to removal of flickering artifacts resulting from interlaced raster scanning.
2. Description of the Prior Art
It has long been recognized that visual displays are a powerful communication tool, particularly for allowing rapid assimilation of information by a viewer. Further, it has been recognized that the utility of computers is limited by the amount of information which can be communicated to a user. Accordingly, the overlaying of display frames and information has become widespread and popular to increase the amount of information which can be displayed within an area limited either by the physical dimensions of the display or the user's field of view. The overlaying of information is generally handled in planes (any of which may include transparent regions to allow viewing of portions of underlying information) which can be rearranged in order along a virtual line of sight of the user. This technique also may be useful in rendering of images of three-dimensional shapes.
The limitations of the available broadcast spectrum as well as the demand for pictorial information has led to the development of additional distribution media such as satellite transmission and cable. Demand for even higher transmission capacity and the degradation of signals by noise has led to digital communication of such pictorial information. Further, the growth of computer network communications has led to compression schemes whereby motion pictures and animation can be transmitted through relatively low bandwidth telephone lines in substantially real time. It can be appreciated that this latter capability requires substantial reduction in data transmission rate, extreme compression and sophisticated error correction and compensation techniques to present an acceptable display. Such compression and digital transmission also greatly increases the transmission capacity of broadband communication links including satellite, microwave and fiber-optic links.
It can be readily understood that virtually any display apparatus will require some degree of quantization of an image. For example, a raster scanned display necessarily requires the image to be dissected into raster lines even though many lines may be provided to preserve a substantial amount of the resolution in the original image. Resolution may thus be quite high even though limited by, for example, the optics and medium used to capture the image (e.g. sensor area or film grain size). If the graphical information is transmitted digitally, there will be additional quantization in luminance and chrominance. Of course, greater resolution in luminance (or chrominance) requires a greater number of bits of luminance (and/or chrominance) information to be transmitted.
These quantizations of the image, separately or in combination, may produce artifacts to which the user may be more or less sensitive. While standards based on practical levels of hardware complexity and bandwidth provide generally acceptable spatial and luminance resolution of a display, increased resolution is constantly sought but is always limited by practical economic factors.
Current practical limitations of data transmission rate and bandwidth have caused the adoption of interlaced raster scans for display of digitally transmitted data on cathode ray tubes and the like. This technique exploits the relatively long persistence/slow negrescence of phosphors and other devices after excitation and allows the refresh rate for the display to be reduced to somewhat below the so-called flicker fusion frequency of the human eye to allow a reduced data transmission rate (one-half the spatial resolution) while generally presenting a display of acceptable appearance. However, the interlaced scan in combination with low refresh rate and either or both forms of spatial and luminance quantization discussed above can form other visual artifacts to which the human eye is quite sensitive given certain image conditions.
In particular, where pixels of substantially differing luminance are provided in the displayed image at corresponding points in adjacent raster lines produced in alternating interlaced fields, the higher luminance pixel will be perceived to flicker at half the frequency of the interlaced fields. The severity of the flickering will vary with the difference in luminance of these vertically (orthogonal to scan line direction) adjacent pixels.
The difference (or gradient over the distance between interlaced scan lines) in luminance substantially equates with contrast in the image. The overlaying of image planes alluded to above generally includes a facility for adjustment of size/dimensions of a window on the display. When a full image frame is displayed in a window of decreased size or the image otherwise downscaled, contrast is usually increased; aggravating the flickering effect.
This artifact is of sufficient irritation to most viewers that numerous filtering techniques which adjust the relative luminance of vertically adjacent pixels have been developed. While all are effective to a degree, many require a trade-off between reduction of the artifact and the engendering of other artifacts or compromise of image fidelity beyond that of quantization since all rely on adjustment of luminance values of one or more of the vertically adjacent pixels from which flickering is observed. Further, practical implementations must consider computational complexity within an environment having severe time constraints imposed by the human flicker fusion frequency.
For example, pages 1543-1544 of an IBM TDB of September, 1987, by J. K. Aggarwal describes a technique of computing a luminance difference, imposing a threshold and, when the threshold is exceeded, scaling the difference value by a correction factor and combining this scaled correction value back into the original luminance value, either additively for the lesser luminance pixel or subtractively for the higher luminance pixel. A graphical depiction of the correction transfer function of such a filter is shown in FIG. 1. Such a filter can engender a visible step artifact at the detection threshold such as at a location where the image may contain a relatively smooth horizontal gradient in luminance over an area.
It should also be recognized that different frame rates have become standards in different countries, such as the United States and Europe. Therefore, flickering will be observed under different image conditions and different correction factors are required even if only a single correction factor is to be applied at a threshold. Therefore, since the correction factor, n, is fixed in FIG. 1, the filter transfer function is not adjustable.
Further, a single threshold and a single correction factor may overcorrect luminance near the threshold and still allow substantial (although somewhat reduced) flicker to be perceived for luminance differences well above the threshold. That is, in order to reduce flicker where a high luminance difference occurs between vertically adjacent pixels, it is necessary to utilize an increased correction factor.
Different correction factors are graphically depicted as filter transfer functions having different slopes in FIG. 2. However, increased correction factors alter the image in a manner which is perceived as increased blurring or loss of delicate graphic features that have a low luminance difference (e.g. near a luminance difference threshold). Thus, if the correction factor of FIG. 1 could be changed to correspond to any one of the correction factors shown in FIG. 2 and used with the single threshold of FIG. 1, adjustability would be provided to accommodate, for example, different frame rates but at the expense of possibly aggravating step luminance artifacts and degrading the image through blurring while still not providing adaptability to different luminance differences between vertically adjacent pixels in the image.
Multiple correction filters may be used to reduce the amount of such blurring of the image. In this case, multiple correction factors are made available as depicted in FIG. 2 and switching between the correction factors is performed at different thresholds as depicted in FIG. 3. Different transfer functions can be derived by the manner in which the corrected value components are combined as depicted by the two solid lines of FIG. 3 approximating different and slightly varying slopes. That is, some degree of overall adaptive non-linearity may be simulated in this manner while the filter transfer function remains piecewise linear. However, such switching also engenders step artifacts at all thresholds since the filter transfer function includes a step discontinuity at each threshold.
Selectivity between filters such as filters corresponding to the solid lines in FIG. 3 can be achieved by evaluation of additional pixels at the cost of additional storage and computational burden. For example, similar luminance above or below the pixels which may cause flicker may be evaluated to determine an edge and a higher correction factor employed while a large difference in luminance value of a small number of pixels may indicate an image detail to which only a small correction factor should be applied.
(It should be noted that while FIG. 1 is designated “PRIOR ART”, FIGS. 2 and 3, while not illustrating the invention, are provided to assist in conveying an understanding of the invention by contrast therewith. Accordingly, no portion of FIG. 2 or 3 is admitted to be prior art in regard to the present invention.)
In summary, no known arrangement provides good suppression of image flicker without significant image degradation. Many known arrangements rely on balancing the effects of a flickering artifact with another type of artifact such as a step artifact which results from the computation of the flicker correction to be applied. Computational and hardware complexity and cost have limited the provision of adjustability and/or adaptive behavior of known flicker filter arrangements and thus do not eliminate flicker under all conditions or accommodate different display standards. Conversely, filter transfer functions of flicker filters of acceptable simplicity and economy generally exhibit discontinuities such as the step discontinuities of FIGS. 1-3 which engender other objectionable artifacts and/or compromise image fidelity while providing differing degrees of flicker reduction.
It should be recognized that the comparison of two vertically adjacent pixel luminance values requires storage of luminance values (either original or corrected) corresponding to an entire interlaced field (one-half of all pixels displayed) as well as a substantial computational burden which must be performed in substantially real time. Such storage capacity can only be achieved with substantial hardware cost even though special purpose chips for such purposes are commercially available. In digital transmission systems capable of processing digital image data, such buffering must be done prior to the digital encoder which is generally integrated with other processing structure (e.g. decompression processor) and access to the digital encoder is thus often very difficult and clumsy, if possible at all.
Further problems are associated with multiple images. One proposal for so-called “picture-in-picture” displays is to double sample both the main image and the small image. However, such an approach would require storage of a full interleaved field of each image and substantial computational overhead which would not contribute to the final displayed image (e.g. the portion of the large image occluded by smaller images). Another suggested approach is described in U.S. Pat. No. 5,850,263 and involves sampling a full frame on a field basis and filtering each field with a different coefficient or correction factor to reduce flickering. However, while possibly reducing computational burden and allowing application of different correction factors, this approach blurs the interlaced fields. Therefore, the image would be of very poor quality if used on computer generated graphics images and only marginally acceptable with video images; requiring different treatment by a programmer or user. Neither of these approaches accommodates multiple image planes.
Accommodation of multiple image planes is considered desirable to the point of being virtually required in a digital image transmission set-top box used to receive and decode a digital transmission and generate signals suitable for application to a television set, sometimes referred to under the MPEG-2 standard as a target decoder (as distinct from various signal processing and decoding elements within it). Multiple image planes are generally necessary in order to provide the programmer or user with the capability of easily and efficiently organizing the display presentation to cleanly add or remove new images and text to the screen without disturbing images that are already being presented or to the selection and editing or rearrangement thereof independently of other images.
Such organization of the display can be performed at the transmitter by a programmer that may also apply a particular form or amount of flicker filtering to each image plane, depending on the source and content thereof. The user may also desire control of arrangement of the display to include multiple images from multiple sources but is generally not involved in the choice of filtering. Separate planes may also facilitate handling of different types of image information. For example, one plane may be dedicated to a video source while other planes may be separately used for text display or other images, either full-size or scaled.
In any case, the programmer knows the origin of the source material used in the various planes and is aware of the need to flicker filter a display that may be expected to flicker. Conversely the programmer would know that a display which would not be expected to flicker should not be flicker filtered in order to maximally preserve image fidelity. The programmer will also have empirical knowledge of the preferred filter to select for each type of image and its source. The flicker filter selection problem is at its worst when the set-top box is being used for Internet access by presenting multiple planes of web-based images and text while simultaneously viewing a downscaled MPEG video stream or still image while a background plane may be showing a video stream or still image that has not been downscaled.
One way of performing flicker filtering in such a case would be to apply a flicker filter to the image of each plane appropriate to the type of image data contained therein. However, such an approach would require double sampling, as alluded to above, for each image plane and storage hardware would become prohibitive to do so for far fewer than the number of planes (e.g. five or more) it is much preferred to provide.
Additionally, as alluded to above, while a processor of substantial power is necessary in a set-top box for decompression and error recovery or concealment in substantially real time, the processing demands of such a function are very substantial and may vary quickly when digital image transmissions are made over a lossy medium such as the Internet. Other hardware is generally provided for digital audio decoding in as many formats as possible, transport demultiplexing of digital audio and video packets, digital to analog encoder (DENC) and graphics features such as line and polygon drawing and fills, text rendering, support for multiple graphics resolutions and support for multiple graphics planes containing images, text, video or cursor in mixed combinations including facilities for windowing, downscaling, zooming and other image manipulations.
Generally an on-chip microprocessor is also included to support user programmable features such as implementation of Internet access, interactive data services, program guides and local on-demand information such as navigation of menus. The on-chip processor is therefore not used for any of the functions noted in the preceding paragraph and should not be used for image enhancement or image artifact removal in order to avoid compromise of the functions for which it is intended.
Therefore, it can be seen that the amount of image processing hardware required to provide features demanded by the market is very extensive and of very substantial cost even when special purpose chips are manufactured in production quantities. Present hardware presents substantial difficulties in implementing flicker filters which may also require extensive hardware for only marginally satisfactory performance. The on-chip microprocessor cannot realistically be used to implement flicker filters and no filtering arrangement is known that can be implemented at an economically viable cost which does not cause a trade-off between the severity of different types of artifacts or which is reasonably effective on all ranges of contrast differences in all types of images and, in particular, images which can be manipulated at the will of the user in a manner which will generally alter the likelihood of production of objectionable flicker effects and the severity thereof.